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FOREWORD 


This  volume  summarizes  an  intensive  in-house  Department  of  Defense  (DoD)  researeh 
and  development  program  performed  by  various  members  of  the  Range  Commanders  Couneil 
Signature  Measurement  Standards  Group  (RCC/SMSG)  Radar  Cross  Seetion  (RCS)  Committee 
(hereafter  referred  to  as  the  Radar  Committee).  The  work  was  led,  managed,  and  eoordinated  by 
the  Signature  Technology  Office,  Air  Force  Research  Laboratory,  Wright-Patterson  Air  Force 
Base,  from  January  1997  to  July  2000.  This  volume  describes  the  execution  of  the  DoD  Radar 
Cross  Section  Demonstration  Program  for  RCS  range  certification.  Organized  around 
International  Standards  Organization  (ISO)  Guide  25  and  its  U.S.  equivalent,  ANZFNCSL  Z- 
540-1-1994  (referred  to  herein  as  Z-540),  this  program  was  designed  to  test  the  feasibility  of 
using  Z-540  to  standardize  the  documentation,  processes,  and  procedures  used  by  all  DoD  RCS 
measurement  facilities.  Using  three  highly  distinct  DoD  RCS  test  facilities  as  test  cases,  the 
DoD  RCS  Demonstration  Program  executed  a  highly  successful  and  realistic  assessment  of 
DoD’s  RCS  measurement  capabilities.  In  addition,  the  program  laid  the  groundwork  for  an 
industrial  RCS  certification  process,  which  closely  mirrors  the  DoD  program.  If  industry  follows 
through  as  planned,  all  U.S.  Government  and  industrial  RCS  measurement  ranges  could  be 
certified  as  early  as  January  2004. 

The  authors  of  this  document  acknowledge  the  many  contributors  to  the  DoD  RCS 
Demonstration  Program  in  Attachment  A  of  this  report. 


Brian  M  Kent,  Ph  D 
Research  bellow 
Member.  RCC  SMSGt Radar) 


Robert  Williams.  Director 
Signature  Technology  OfTice 
Air  Force  Research  l.aboratoiv 


J<>hn  l.euis.  Chair 

Range  Commanders  Council  -  Signature 
.Measurements  and  Standards  Group  (Radar  Committee) 
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EXECUTIVE  SUMMARY 


In  January  1997,  the  Range  Commanders  Council  Signature  Measurement  Standards  Group 
(RCC/SMSG),  in  coordination  with  the  RCC  Taskmaster,  established  RCC/SMSG  Task  #08 
titled,  “DoD  RCS  Certification  Feasibility  Demonstration  for  Static  and  Dynamic  RCS 
Measurement  Facilities.”  Referred  to  herein  as  the  DoD  RCS  Demonstration  Program,  the  task 
assessed  the  feasibility  of  establishing  common  documentation  standards  to  describe  and  certify 
the  operation  and  capabilities  of  the  DoD’s  diverse  RCS  measurement  facilities  located 
throughout  the  United  States.  Through  the  mutual  efforts  of  the  United  States  Air  Force  (USAF) 
46  Test  Group,  in  cooperation  with  the  RCC/SMSG  Radar  Committee,  the  demonstration 
program  described  herein  was  entirely  successful  and  should  lay  the  groundwork  for  similar 
technical  or  laboratory  calibration  certification  efforts  throughout  the  DoD  RCS  measurement 
community. 

More  specifically,  this  report  describes  how  Z-540  was  applied  in  a  DoD  demonstration 
program  to  organize  RCS  range  documentation.  This  report  is  a  follow-up  to  the  DoD  RCS 
Demonstration  Program  recently  presented  to  the  Antenna  Measurement  Techniques  Association 
[2].  In  June  2000,  the  RCC/SMSG  certified  that  two  dynamic  RCS  measurement  facilities,  the 
Air  Force  Research  Laboratory’s  (AFRL’s)  Advanced  Compact  Range  (ACR)  and  the  Naval  Air 
Warfare  Center-Aircraft  Division’s  (NAWC-AD’s)  Atlantic  Test  Range  (ATR)  met  the  Z-540 
documentation  standards  established  by  the  DoD  RCS  Demonstration  Program.  Since  AFRL 
plans  to  unilaterally  require  mandatory  Z-540  compliance  for  DoD  contractors  performing  RCS 
measurements  with  AFRL  after  1  January  2004,  the  RCS  certification  review  process  described 
in  this  report  will  be  the  model  used  for  industrial  compliance.  After  reviewing  the  Z-540 
standard,  this  report  will  summarize  the  certification  review  process,  and  discuss  the  outcomes, 
results,  and  lessons  learned  from  the  DoD  RCS  Demonstration  Program  from  the  perspective  of 
the  ranges  reviewed,  as  well  as  the  volunteer  range  book  reviewers. 


CHAPTER  1 


INTRODUCTION  TO  THE  ANSI/NCSL  Z-540-1-I994  STANDARD 


I.I  General 

This  chapter  presents  a  brief  overview  of  the  American  National  Standards 
Institute/National  Calibration  Standards  Laboratory  ANSI/NCSL  Z-540-1-1994  Standard  [1] 
(referred  to  herein  as  Z-540).  The  Z-540  offers  a  straightforward  way  to  organize  radar  cross 
section  range  (RCS)  documentation.  We  begin  by  discussing  the  major  points  of  Z-540  and  how 
to  organize  a  format-universal  RCS  Range  Book.  Since  Z-540  is  the  U.S.  equivalent  of 
International  Standard  Organization  (ISO)  25,  it  is  especially  useful  for  two  reasons:  (1)  it  is 
applicable  to  RCS  ranges ;  and,  (2)  its  quality  control  requirements  are  consistent  with  the  ISO 
9000  series  of  quality  standards.  Properly  applied,  Z-540  will  greatly  improve  the  quality  and 
consistency  of  RCS  measurements  produced  by  RCS  measurement  facilities  and  reported  to 
range  and  DoD  customers. 

The  signature  measurement  community  is  aware  of  the  complexities  of  a  modem  RCS 
measurement  range.  Electromechanical  systems,  regardless  of  range  type,  must  contain  a  myriad 
of  hardware  and  software  to  operate  a  range  within  intended  design  parameters.  Naturally, 
operating  a  range  as  designed  helps  assure  that  it  produces  consistently  high-quality  data.  Often, 
the  job  of  keeping  current  range  documentation  may  seem  overwhelming,  especially  since  there 
are  currently  no  standards  or  guidelines  for  organizing  such  documentation. 

The  initial  purpose  of  RCC/SMSG  Task  #08,  a  research  activity  involving  the  Air  Force 
Research  Laboratory  (formerly  Wright  Laboratory),  the  Naval  Air  Warfare  Center- Aircraft 
Division  Chesapeake  Test  Range,  and  the  Electromagnetic  Fields  Division  of  the  National 
Institute  of  Standards  and  Technology  (NIST)  is  described  herein.  A  top-level  overview 
schedule  of  Task  #08  is  shown  in  Figure  1-1.  This  task  and  associated  research  was  designed  to 
improve  the  quality  and  repeatability  of  RCS  data.  One  significant  side  contribution  of  this 
project  has  been  the  development  and  implementation  of  highly  accurate  and  repeatable  static 
RCS  range  calibration  standards  [3,4].  A  second  benefit  has  been  a  significant  improvement  in 
the  sharing  of  technical  information  and  measurement  techniques  between  the  various  industrial 
and  government  RCS  ranges.  The  annual  RCC/SMSG  RCS  certification  conference  technical 
exchange  has  tmly  inspired  the  RCS  industry  to  work  together  in  a  more  cooperative 
environment. 
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Figure  1-1.  Top-Level  Final  Task  Status  Report  RCC/SMSG  Task  #08. 


The  activity  reported  herein  also  focuses  on  the  establishment  and  review  of  an  RCS 
range  documentation  standard,  which  includes  processes,  procedures,  quality  control,  and 
characterization  information.  After  closely  examining  the  technical  literature  and  interviewing 
many  ranges  during  the  1995-1997  time  frame,  the  RCC/SMSG  Radar  Committee  found  that 
there  is  no  standard  method  for  creating  and  maintaining  range  documentation.  Although  some 
ranges  had  excellent  and  thorough  documentation  [4],  other  ranges  could  clearly  benefit  from 
improving  and  organizing  their  present  documentation  set.  The  Radar  Committee  also  found  that 
the  universal  availability  of  the  range  documentation  to  the  range  operating  staff  and  customers 
varied  greatly  from  range  to  range.  Clearly,  the  operating  staff  needed  the  most  up-to-date  and 
thorough  information  available  in  order  to  produce  consistently  high  quality,  repeatable  RCS 
data. 


During  the  process  of  examining  various  candidates  for  an  RCS  documentation  standard, 
several  factors  were  considered.  First,  from  a  DoD  perspective,  the  Radar  Committee  examined 
typical  military  specifications  and  standard  federal  contract  report  deliverables  to  see  if  such 
standards  could  be  exploited  and  customized  for  the  documentation  of  RCS  ranges.  In  light  of 
the  new  DoD  5000  series  of  federal  acquisition  regulations,  we  are  seeing  significant  federal 
procurement  and  acquisition  reform.  In  the  future,  DoD  will  quickly  move  away  from  the  use  of 
rigid  military  specifications  (Mil  Specs)  in  federal  contracts  in  favor  of  more  universally 
available  and  understood  “commercial  grade  standards.”  Because  future  DoD  contracts  will  rely 
less  on  Mil  Specs,  they  were  ruled  out  as  a  possible  RCS  range  standard  model. 

The  Radar  Committee  also  examined  the  ongoing  work  of  the  Institute  of  Electrical  and 
Electronics  Engineers  Antennas  and  Propagation  Society  (IEEE- APS)  Committee  on  RCS,  of 
which  the  National  Institute  of  Standard  Technology  (NIST)  is  an  active  participant.  While  this 
important  committee’s  work  was  not  complete  in  1997  (and  still  is  not  complete),  it  is  clear  that 
the  IEEE- APS  will  not  focus  on  range  documentation  standards. 
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Finally,  the  RCC/SMSG  Radar  Committee  examined  ongoing  work  in  the  European 
Community  (EC),  which  already  mandates  most,  if  not  all,  the  ISO  9000/9002  series  of 
regulations  for  their  commercial  business  transactions.  While  ISO  9000  by  itself  is  not 
applicable  to  RCS  ranges,  the  technical  standard  ISO  25  appeared  to  satisfy  most  of  the 
requirements  for  a  quality-oriented  RCS  range  documentation  system.  We,  therefore,  looked 
into  the  US  equivalent  of  ISO  25,  which  is  Z-540. 

We  postulated  in  the  beginning  of  this  program  that  standardized  range  documentation, 
universally  available  to  staff  and  customers  alike,  would  greatly  benefit  the  RCS  measurement 
community.  Eurther,  it  is  the  view  of  the  RCC/SMSG  Radar  Committee  that  Z-540  [1,  5]  with 
clarification  [6]  is  the  appropriate  and  applicable  guide  for  RCS  ranges. 

1.2  A  Closer  Look  at  ANSI/NCSL  Z-540-1-1994 

ANSENCSE  Z-540-1-1994  [1]  is  available  for  a  nominal  fee  from  NCSE  (tel.  #  303-440- 
3339).  The  title  of  the  standard  is  Calibration  Eaboratories  and  Measuring  and  Test  Equipment  - 
General  Requirements,  and  it  provides  a  framework  for  collecting  and  maintaining  information 
to  support  a  quality-oriented  system  of  documentation  for  any  range  that  produces  calibrated 
data.  In  essence,  the  standard  is  very  general,  and  could  easily  accommodate  many  types  of 
calibrated  scientific  measurements  (e.g.,  voltage,  lumens,  temperature,  etc.).  Basically  a  13-page 
guide,  Z-540  summarizes  the  general  documentation  requirements  for  any  facility  producing 
quality  calibrated  data.  Prior  to  outlining  Z-540,  we  would  like  to  point  out  that  an 
accompanying  guide  has  been  written  [6],  which  helps  the  reader  interpret  Z-540  with  respect  to 
RCS  ranges  specifically. 

Z-540  is  organized  into  two  parts.  Part  I  is  preceded  by  a  three-section  introduction, 
which  includes  references  and  standard  definitions.  The  definition  section  is  especially  helpful 
in  establishing  a  common  dialog  for  the  remainder  of  the  standard.  The  main  thrust  of  the 
standard  is  Part  I  itself  titled,  “General  Requirements  for  the  Competence  of  Calibration 
Eaboratories.”  It  consists  of  13  subsections  outlining  the  basic  standard  and  the  information 
needed  for  a  range  to  document  to  the  standard.  Part  II  of  the  standard,  “Quality  Assurance 
Requirements  for  Measuring  and  Test  Equipment,”  pertains  to  the  control,  usage,  and  calibration 
of  individual  pieces  of  equipment  used  to  produce  calibrated  data.  Part  I  is  the  most  applicable 
and  appropriate  portion  of  this  standard  when  applied  to  the  specific  case  of  RCS  ranges.  Part  II 
was  not  used  in  any  part  of  RCC/SMSG  Task  #08. 

1.3  Using  Z-540  to  Establish  an  RCS  Range  Quality  System 

The  central  theme  of  Z-540  is  the  establishment  and  maintenance  of  an  active  and 
ongoing  Measurement  Quality  Assurance  Program  (MQAP).  Most  ranges  already  incorporate 
measurement  assurance  as  part  of  their  normal  range  operations,  although  this  phrase  is  rarely 
quantified.  However,  it  may  be  helpful  to  briefly  define  the  elements  of  an  MQAP  [6,  7]. 

A  typical  MQAP  consists  of  documented  technical  requirements  designed  to  insure 
repeatable  RCS  performance  from  a  given  measurement  range.  The  MQAP  documents 
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measurement  procedures,  data  processing  procedures,  range  uncertainty,  and  error  estimation 
analysis,  while  simultaneously  identifying  a  sound  management  structure  set  up  to  efficiently 
operate  and  maintain  the  RCS  range.  It  also  strongly  encourages  an  active  inter-laboratory 
comparison  program  (within  reasonable  time  and  range  availability  constraints)  and  an  ongoing 
research  program  to  improve  RCS  data  quality.  At  first  reading,  it  may  appear  that  standard  Z- 
540  actually  implies  the  development  of  a  separate  MQAP  in  addition  to  the  documentation 
required  by  the  standard.  Actually,  completion  of  the  documentation  required  by  Z-540  will,  in 
effect,  establish  a  documented  MQAP. 

1.4  A  Section  Overview  of  Z-540  -  Part  I 

Z-540  opens  with  a  three  section  “Foreword,”  which  defines  the  scope  of  the  standard 
(Section  1),  important  references  (Section  2),  and  a  set  of  excellent  baseline  definitions  (Section 
3).  These  sections  all  help  the  reader  define  a  common  frame  of  reference  for  interpreting  the 
standard.  After  the  Foreword,  Part  I  (General  Requirements  for  the  Competence  of  Calibration 
Laboratories)  presents  the  details  of  the  standard  itself.  Part  I  consists  of  13  individual  sections 
(Sections  4-16),  which  present  an  ideal  organization  for  a  range  documentation  set  that  parallels 
each  section  of  the  Z-540  standard.  Let  us  review  each  section  by  title,  and  provide  some  insight 
into  their  purposes.  Later,  we  will  define  how  each  item  is  evaluated  in  the  certification 
program. 

Z-540  Section  4  (Organization  and  Management)  briefly  describes  how  to  document  the 
organization  performing  the  calibrations.  Using  an  open  format,  the  organization  and  personnel 
would  be  identified  along  with  their  chain  of  command.  This  section  also  identifies  the  technical 
and  quality  leaders  responsible  for  the  technical  and  quality  aspects  of  range  operations. 
(Naturally,  in  smaller  RCS  ranges,  these  two  positions  may  be  held  by  the  same  person.)  Z-540, 
Section  4,  requires  that  all  range  individuals  (and  any  alternates)  be  identified.  Also,  Section  4 
requires  that  the  RCS  range  be  organized  and  operated  in  such  a  way  as  to  assure  that  RCS  data 
confidence,  independent  technical  judgment,  and  data  integrity  are  always  maintained. 

Z-540  Section  5  (Quality  System,  Audit,  and  Review)  is  one  of  the  three  major  sections 
of  Z-540  and  sets  the  stage  for  the  remainder  of  the  document.  It  says  that  the  range  will 
establish  and  maintain  an  active  “quality  system”  with  appropriate  internal  audits  and  reviews 
suitably  “appropriate  to  the  type,  range,  and  volume  of  calibration  activities  it  maintains.”  This 
sentence  is  very  important  as  it  distinguishes  Z-540  and  ISO  25  from  the  overall  ISO  9000/9002 
series  in  the  important  phrase  “appropriate  to  the  type,  range,  and  volume  of  calibration 
activities. . . ”  Subsection  5.2  is  more  specific  and  lays  out  the  elements  that  should  be  included  in 
the  quality  manual.  Some  examples  include:  (5.2-d)  procedures  to  control  and  maintain 
documentation,  (5.2-g)  procedures  for  achieving  traceability,  (5.2-h)  scope  of  calibration  and 
verification  activities,  (5.2-k)  procedures  for  handling  (calibration)  devices,  (5.2-1)  references  to 
major  equipment  and  reference  measurement  standards,  (5.2-n)  references  to  quality  assurance 
practices,  including  inter-laboratory  comparisons,  proficiency  testing,  and  internal  quality  control 
practices,  (5.2-o)  procedures  for  corrective  action  when  RCS  data  discrepancies  are  found,  and 
(5.2-q)  procedures  for  dealing  with  customer  complaints.  Although  Sections  5.3  -  5.6  describe  a 
few  additional  details  on  the  audit  and  review  process,  the  creation  of  the  range-appropriate 
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“quality  manual”  in  Section  5.2,  as  it  pertains  specifically  to  RCS  measurement  ranges,  is  the 
main  intent  of  standard  Z-540. 

Z-540  Section  6  (Personnel)  describes  how  a  range  defines  and  maintains  the  appropriate 
staff  needed  to  operate  the  RCS  range.  It  also  requires  ranges  to  describe  their  initial  and 
ongoing  training  programs  used  to  maintain  the  technical  competence  of  their  operating  staffs. 
This  information  should  be  easy  to  assemble,  as  most  organizations  routinely  obtain  and  manage 
this  type  of  information. 

Z-540  Section  7  (Accommodation  and  Environment)  requires  a  range  to  describe  its 
environmental  “accommodation”  and  to  identify  what  environmental  factors  may  affect 
measurement  results.  The  latter  category  may  be  very  short  for  some  indoor  ranges  but  lengthy 
for  outdoor  or  dynamic  measuring  applications.  For  instance,  an  outdoor  static  range  may  wish 
to  record  temperature,  winds,  precipitation,  inversions,  or  other  phenomena  known  to  affect  the 
RCS  measurement  data.  When  the  effects  cannot  be  quantified  (in  RCS  error  terms),  it  is  still 
very  important  to  identify  and  record  when  such  conditions  occurred. 

Z-540  Section  8  (Equipment  and  Reference  Materials)  describes  how  equipment, 
maintenance,  and  reference  materials  and  records  are  tracked.  Again,  most  organizations  have 
systems  to  control  and  identify  all  equipment  used  by  a  typical  range.  By  systematically 
organizing  this  information  into  a  “range  appropriate”  configuration  control  document,  one 
makes  a  good  external  record  of  the  measurement  system,  as  well  as  an  excellent  and  useful 
internal  document  valuable  to  the  range  technicians  and  engineers.  For  more  complex  ranges 
(those  ranges  having  more  equipment  and  subsystems),  this  section  becomes  exceedingly 
important  in  managing  the  overall  health  and  operational  status  of  the  individual  pieces  of 
equipment.  In  the  DoD  RCS  Demonstration  Program,  the  dynamic  RCS  measurement  system  at 
Atlantic  Test  Range  (ATR)  consisted  of  hundreds  of  pieces  of  equipment.  It  is  no  surprise  that 
the  ATR’s  Section  8  was  a  significant  portion  of  their  total  range  book  page  count.  Smaller  RCS 
range  systems  would  have  a  correspondingly  smaller  Section  8. 

Z-540  Section  9  (Measurement  Traceability  and  Calibration)  and  Section  10  (Calibration 
Methods)  together  define  the  information  needed  to  establish  traceability  to  “national, 
international,  or  intrinsic  standards  of  measurement.”  At  RCS  ranges,  this  is  usually 
accomplished  through  indirect  “ratio  or  reciprocity  type  measurements  (9. 3-d),”  wherein  an 
unknown  target  is  measured,  a  known  RCS  standard  is  measured,  and  a  ratio  is  computed  with 
suitable  adjustments  using  a  highly  accurate  numeric  RCS  estimate  of  the  known  calibration 
target.  Though  every  range  performs  calibrations  in  a  slightly  different  manner,  Z-540  requires 
that  such  procedures  be  rigorously  documented,  tested,  and  verified,  whenever  possible,  with 
other  RCS  ranges.  One  of  the  most  important  aspects  of  these  two  sections  is  contained  in 
paragraphs  10.2  -  10.4,  which  describe  the  need  for  written  procedures  to  execute  appropriate 
calibrations.  Once  again,  having  this  procedural  information  universally  available  to  your  range 
staff,  including  archival  results  (e.g.,  what  the  correct  calibration  measurements  should  be)  will 
greatly  improve  the  quality  and  repeatability  of  RCS  calibration  values.  In  addition,  specific 
calibration  processes  selected  for  use  by  a  given  range  must  be  routinely  tested.  Calibration 
procedures  discussed  in  [2,3]  are  one  way  of  meeting  this  requirement,  though  other  documented 
methods  may  be  used  as  well. 


1-5 


Z-540  Section  1 1  (Handling  of  Calibration  Items)  is  self-  explanatory  and  essentially 
states  the  need  for  maintaining  physical  control  over  calibration  items  used  to  produce  calibrated 
RCS  range  data.  Such  items  should  be  appropriately  stored  and  protected  when  not  being  used. 

In  addition,  it  points  out  the  need  to  uniquely  identify  each  primary  range  calibration  target,  so 
there  is  never  confusion  regarding  the  use  of  one  calibration  target  over  another. 

Z-540  Section  12  (Records)  is  also  self  explanatory  and  basically  states  that  the  range 
must  have  a  record  keeping  system  consistent  with  “its  particular  circumstances.”  It  also 
specifically  states  that  the  record  keeping  system  should  be  consistent  with  the  overall  range 
quality  system. 

Z-540  Section  13  (Certificates  and  Reports)  describes  the  elements  that  must  appear  in  a 
“range  certificate,”  otherwise  called  an  RCS  data  or  test  report.  It  emphasizes  that  each  range 
should  have  a  minimum  subset  of  information  in  the  report,  including  range  identity  (title,  name 
and  address  of  range),  customer  identity  (name  and  address  of  customer),  description  of  the 
calibration  item  (target),  dates  of  the  calibration  measurements,  type  of  process,  calibration 
techniques  used,  and  so  on.  It  is  meant  to  provide  consistency  of  data  reporting  from  a  specific 
range,  as  well  as  to  establish  more  universal  reporting  standards  from  range  to  range. 

Z-540  Sections  14  (Subcontracting  of  Calibration)  and  Section  15  (Outside  Support 
Services  and  Suppliers  That  Affect  Calibration  Results)  are  both  brief  but  important.  In  essence, 
they  state  that  if  outside  suppliers  furnish  any  of  the  calibrated  RCS  data,  they  are  subject  to  the 
requirements  of  the  Z-540  standard,  provided  the  prime  or  supplying  range  conforms  to  this 
standard. 

Finally,  Z-540  Section  16  (Complaints)  assures  that  the  range  has  a  formal,  quality- 
oriented  process  for  responding  to  customer  complaints  regarding  any  aspect  of  range  operation. 
The  best  place  to  put  this  documented  policy  is  in  the  quality  manual  (5.2q). 

1.5  Organizing  the  RCS  Range  Book  to  Comply  with  Z-540 

The  following  paragraphs  provide  guidelines  for  the  construction  of  the  RCS  Range  Book 
used  to  document  range  characteristics  in  accordance  with  Z-540.  The  goal  of  these  guidelines  is 
to  maintain  consistency  in  the  layout  of  the  documentation  from  range  to  range.  Volume  I  of  this 
document  provides  a  detailed  description  of  the  layout  of  a  typical  range  book,  so  only  the 
general  points  will  be  repeated  here. 

The  main  goal  of  the  range  book  is  to  organize  the  RCS  system  documentation  into  a 
format  independent  of  those  used  by  the  ranges  performing  the  measurements.  However,  within 
each  section,  company  or  local  documentation  formatting  is  strongly  encouraged  whenever 
possible.  In  this  manner,  substantial  costs  for  reformatting  information  to  Z-540  is  avoided  to 
the  extent  possible,  while  at  the  same  time  affording  a  common  document  layout.  The  goal  here 
is  to  create  a  "living-breathing  document"  which  is  useful  to  the  range  staff  it  was  created  to 
serve.  It  is  the  range  book  that  is  reviewed  technically  by  a  third  party,  and  it  is  the  basis  for 
granting  a  range  certification. 
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The  range  book  sections  will  closely  parallel  the  Z-540  standard,  with  few  exceptions. 
However,  it  is  emphasized  that  the  importance  of  the  range  book  is  in  its  basic  utility  as  a  general 
reference  standard  for  range  personnel  and  customers  alike.  Thus,  the  utility  of  the  range  book 
will  largely  depend  on  the  currency  and  relevance  of  the  information  in  it  and  the  amount  of 
effort  expended  to  create  it. 

Depending  on  the  size  and  type  of  range,  the  range  book  may  be  physically  small  or 
large.  For  instance,  we  would  expect  the  range  book  for  a  highly  complex  dynamic  signature 
measurement  range  to  be  substantially  larger  than  that  of  a  relatively  simple  indoor  far  field 
range.  The  ranges  should  carefully  consider  whether  to  produce  their  range  books  in  hardcopy  or 
electronic  format.  While  a  conventional  paper  copy  would  certainly  meet  all  the  requirements  of 
Z-540,  an  electronic  version  available  to  every  range  employee  on  a  network  server,  an  internal 
or  external  web  page,  or  on  a  permanent  CD-ROM  may  be  a  better  option  in  the  long  run. 
Normally,  if  multiple  hard  copies  of  the  range  book  are  created,  each  must  be  separately  updated 
to  reflect  changes  in  the  range  information  over  a  natural  lifetime.  In  the  case  of  an  electronic  or 
web  page  version,  only  one  copy  (on  the  web  page)  would  need  to  be  updated  and  redistributed. 
Thus,  maintaining  current  documentation  electronically  should  prove  easier  than  maintaining 
paper  copies.  In  either  case,  the  reader  is  encouraged  to  use  whatever  system  is  available  or 
appropriate  for  their  particular  situation. 

The  range  book  (paper  or  electronic)  may  be  the  most  readily  available  range  reference 
for  use  by  any  range  staff  member.  It  is  also  the  documentation  reviewed  by  the  third  party 
certification  review  committee,  as  discussed  later  in  this  report.  Once  completed,  it  is  the 
responsibility  of  each  range  to  keep  its  range  book  current. 

1.6  RCS  Range  Registrations  and  Third  Party  Certification  Review 

Compilation  of  the  range  book  completes  an  important  phase  in  the  formal 
documentation  of  range  performance.  Even  if  a  range  does  not  proceed  to  third  party 
certification,  such  a  document  set  will  greatly  help  sustain  quality  calibrated  RCS  measurements. 
Assuming  an  RCS  range  completes  a  range  book,  the  book  first  needs  to  be  reviewed  internally 
within  the  company  or  government  organization  that  created  it.  This  process  is  called  an  internal 
audit  or  “self  registration,”  using  the  vernacular  of  ISO.  Once  a  range  book  is  “self-reviewed” 
internally  and  checked  against  the  published  evaluation  criteria  [see  Vol.  I  of  this  document], 
range  management  normally  endorses  the  range  book  in  Section  1 .  Their  signatures  attest  that 
management  has  approved  the  range’s  overall  quality  assurance  program  and  its  accompanying 
range  book. 

The  next  part  of  the  certification  process  is  the  independent  third  party  review  committee. 
Z-540,  Section  1.3,  states,  “the  role  of  the  purchaser  in  monitoring  a  supplier’s  (range) 
compliance  with  the  requirements  of  this  standard  may  be  fulfilled  by  a  third  party,  such  as  a 
certification  board.”  As  DoD  strives  to  test  and  demonstrate  the  applicability  of  Z-540,  it  will 
rely  on  an  all-volunteer  certification  board  composed  of  technical  specialists  drawn  from 
government,  academia,  industry,  and  the  National  Institute  of  Standards  and  Technology.  All 
range  book  reviewers  are  volunteer  technical  experts  active  in  the  RCS  measurement  business. 
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Such  a  third  party  review  committee  “certifies”  that  a  range  complies  with  the  Z-540  standard  as 
it  pertains  to  the  specific  RCS  measurement  range. 

A  third-party  review  committee  can  never  guarantee  that  a  “certified”  range  will  not 
make  errors.  We  believe  that  since  Z-540  is  extremely  process-oriented,  proper  documentation 
of  normal  range  processes  and  procedures  will  produce  a  quality  system  that  greatly  reduces  the 
occurrence  of  data  problems  related  to  poor  procedural  execution.  The  Z-540  standard  is  an 
excellent  approach  to  organize,  maintain,  and  present  RCS  range  information.  The  RCS  Range 
Book  should  be  very  useful  to  the  range  and  range  customer  alike.  Assuming  that  a  well- 
managed  RCS  range  has  much  of  the  required  documentation  in  hand,  complying  with  the  Z-540 
standard  should  be  a  gradual,  transitional  process  as  the  range  book  is  formalized.  If  omissions 
are  found  in  the  RCS  range’s  documentation,  the  process  of  implementing  a  measurement  quality 
assurance  program  (MQAP)  as  part  of  the  Z-540  compliance  process  should  fill  those  omissions, 
thereby  improving  the  quality  and  consistency  of  RCS  range  data  supplied  to  external  customers. 
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CHAPTER  2 


THE  DOD  RCS  DEMONSTRATION  PROGRAM  -  AFRL  PERSPECTIVE 


2.1  General 

The  following  paragraphs  describe  the  results  of  the  Air  Force  Research  Laboratory 
(AFRL)  and  the  Naval  Air  Warfare  Center-Aircraft  Division  (NAWC-AD)  joint  DoD  RCS 
Demonstration  Program,  which  was  designed  to  improve  the  quality  and  repeatability  of  RCS 
measurement  data  taken  on  any  government  or  industrial  RCS  measurement  facility.  This 
chapter  will  address  activities  at  AFRL’s  Advanced  Compact  Range  (ACR)  and  Chapter  3  will 
focus  on  the  certification  activities  at  NAWC-AD’s  Atlantic  Test  Range  (ATR). 

As  mentioned  in  Chapter  1,  RCS  measurement  systems  represent  extremely  complex 
electromechanical  systems  containing  a  myriad  of  complex  hardware  and  software  subsystems. 
To  maintain  a  high  level  of  confidence  in  RCS  data  produced  by  these  ranges,  a  quality 
assurance  program  was  established  in  1997.  The  RCC/SMSG  co-sponsored  the  AFRL  and 
NAWC-AD  joint  project  to  test  and  demonstrate  a  single  standard  for  documenting  the 
procedures  and  processes  for  the  three  distinct  DoD  RCS  ranges  shown  in  Figure  2-1.  Officially 
adopted  by  the  RCC  as  RCC/SMSG  Task  #08,  the  DoD  RCS  Demonstration  Program  began  by 
establishing  a  customized  Z-540  RCS-based  quality  assurance  documentation  handbook  [1,  5,  6]. 
Though  this  document  provided  a  context  for  the  documentation  standard,  it  provided  few  details 
regarding  the  evaluation  criteria  and  certification  review  processes  needed  to  complete 
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Figure  2-1.  DoD  RCS  Demonstration  Program  Ranges. 
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RCS  certification.  The  remainder  of  this  report  describes  the  establishment  and  use  of  objective 
evaluation  criteria,  the  certification  review  process,  and  the  overall  success  of  the  two  ranges  at 
AFRL  and  NAWC-AD,  which  have  successfully  fulfilled  the  certification  review  process  to  date. 
(The  third  range,  46TG-RVUMS  at  Holloman  Air  Force  Base  (HAFB)  in  New  Mexico,  is 
undergoing  certification  review,  and  the  review  should  be  complete  by  December  2000.) 

2.2  Defining  Z-540  RCS  Certification 

To  avoid  repeating  the  background  information  in  Chapter  1,  we  begin  by  precisely 
defining  “RCS  certification.”  From  [6,7]: 

“RCS  certification,  as  used  in  the  DoD  demonstration  program,  is  hereby 
defined  as  having  successfully  completed  a  detailed  review  of  an  RCS 
Range  Book  by  a  peer  review  committee  using  published  evaluation 
criteria.” 

The  detailed  review  is  essentially  a  compliance  assessment  that  a  range  has  met  the  relevant 
quality  assurance  documentation  standards  for  Z-540.  In  the  DoD  RCS  Demonstration  Program, 
certification  is  a  review  committee’s  positive  assessment  of  compliance  with  the  Z-540  standard 
and  an  endorsement  of  the  range’s  measurement  processes.  RCS  certification  is  NOT  a  rating 
system  or  a  specific  review  committee  endorsement  of  day-to-day  quantitative  capabilities 
(accuracy,  uncertainty,  and  so  forth). 

To  better  understand  certification  contextually,  let’s  review  each  major  element  of  the 
certification  statement.  We  begin  by  defining  the  RCS  Range  Book.  From  [6],  an  "RCS  Range 
Book”  is  a  detailed  technical  and  quality  assurance  document  written  in  accordance  with  the 
Z-540  standard  by  the  range  to  be  certified.  The  range  book  thoroughly  describes  all  aspects  of 
the  RCS  measurement  range.  Its  20  distinct  sections  describe  every  major  aspect  of  range 
operations,  procedures,  calibration,  and  equipment.  The  range  book  also  explains  how  the  range 
is  organized  and  operated  on  a  day-to-day  basis.  The  range  book  is  meant  to  be  a  ready  daily 
reference  for  the  range  staff,  as  well  as  a  document  a  potential  customer  could  review  prior  to 
using  the  RCS  facility  for  a  test. 

The  “published  evaluation  criteria”  [see  Vol.  I  of  this  document]  are  a  set  of 
approximately  140  specific  written  evaluation  criteria  that  the  range  book  must  fulfill  to  become 
RCS  certified.  The  evaluation  criteria  provide  a  complete  listing  of  the  specific  minimum 
information  required  for  each  of  the  20  range  book  sections.  Since  the  range  knows  the 
evaluation  criteria  up  front,  they  provide  a  definitive  guide  for  the  range  staff  to  help  prepare  the 
range  book  material.  The  evaluation  criteria  are  also  used  by  the  range  to  “self  review”  or 
“register”  the  range  book  prior  to  certification  review  by  a  third  party. 

The  “peer  review  committee  ”  consists  of  three  volunteer  RCS  experts  from  government, 
industry,  and/or  academia  who  review  the  RCS  Range  Book  for  compliance  with  the  Z-540 
standard  using  the  evaluation  criteria  mentioned  above.  The  volunteer  reviewers  provide 
feedback  to  the  range  on  non-compliant  range  book  sections  and  work  with  the  range  in  question 
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to  help  revise  the  range  book  and  make  it  fully  compliant  with  the  standard.  Literally,  the  review 
committee  is  there  to  help  the  range  complete  the  certification  process.  Note  that  ACR  and  ATL 
were  solely  reviewed  by  RCS  experts  employed  outside  DoD  to  avoid  any  perceived  conflict  of 
interest.  (The  decision  to  use  external  range  book  reviewers  employed  outside  DoD  was 
coordinated  and  approved  by  the  RCC.)  The  ongoing  46TG  RMS  VHF/UHF  Measurement 
System  (RVUMS)  certification  review  also  employs  non-DoD  reviewers.  All  range  book 
reviewers  must  meet  minimum  qualifications  set  by  the  RCC/SMSG  Radar  Committee,  and  must 
be  assigned  to  a  specific  review  committee  by  the  Chair  of  the  RCC/SMSG.  As  a  courtesy,  the 
names  and  current  employers  of  potential  range  book  reviewers  are  provided  to  the  RCC/SMSG, 
who  carefully  screen  potential  reviewers  for  financial  or  programmatic  conflicts  of  interest  If  an 
apparent  conflict  of  interest  is  identified  by  the  RCC/SMSG,  the  potential  reviewer  is  assigned  to 
another  certification  review  team  where  no  such  conflict  exists. 

2.3  The  Detailed  Z-540  RCS  Certification  Process 


The  RCS  certification  review  process  created  by  the  RCC/SMSG  may  seem  complicated 
at  first  glance,  but  it’s  actually  a  manageable,  easy-to-follow,  step-by-step  recipe.  To  illustrate 
the  overall  review  process,  refer  to  Figure  2-2.  The  process  begins  with  the  “Registered”  Phase 
when  a  range  decides  to  become  “registered”  to  the  Z-540  standard.  This  involves  generating  a 
range  book  (paper  or  electronic  ),  which  meets  the  published  evaluation  criteria  [see  Vol.  I  of  this 
document]  for  RCS  certification.  For  most  ranges,  this  process  can  take  6-24  months  depending 
on  the  complexity  of  the  RCS  range  measurement  system  and  the  range’s  initial  state  of 
documentation.  If  a  range  has  organized  documentation  and  thoroughly  documented 
measurement  and  calibration  procedures  available  at  the  onset,  the  process  can  be  quickly 
completed.  If  these  documents  need  to  be  created  by  the  range  staff,  the  process  obviously  takes 
longer.  Once  the  range  book  draft  is  generated,  the  range  performs  an  internal  “self  review”  of 
the  range  book,  using  the  same  evaluation  criteria  used  by  the  third  party  review  committee. 

Once  this  “self  review”  is  successfully  completed,  the  range’s  designated  quality  manager 
normally  endorses  the  range  book  in  Section  1  (Introduction  and  Endorsements).  Once  reviewed 
and  signed  internally,  a  range  is  considered  “Registered  to  Z-540.”  A  range  must  become  RCS 
“registered”  prior  to  requesting  a  third  party  certification  review,  a  practice  consistent  with  ISO 
standard  review  procedures. 

The  next  step  in  the  certification  process  is  the  Notification  Phase.  A  range  desiring 
certification  contacts  the  Chair  of  the  RCC/SMSG  (as  of  the  date  of  this  publication,  Mr.  Dale 
Bradley,  USAF  Arnold  Engineering  Development  Center,  tel.  #  931-454-4242,  e-mail: 
dale.bradley  @  amold.af.mil) .  The  range  requests  the  formation  of  an  RCS  Certification  Review 
Committee.  Within  30  days  of  being  contacted,  the  Chair  assigns  a  third  party  committee  of 
three  RCS  experts  to  review  the  range  book  in  question.  This  third  party  committee  consists  of 
experts  in  RCS  recommended  by  the  RCC/SMSG  Radar  Committee.  As  stated  earlier,  the 
RCC/SMSG  Chair  assigns  the  review  committee  members  to  assure  there  is  no  perceived 
conflict  of  interest  between  the  reviewers  and  the  range.  Once  the  review  committee 
membership  has  been  named,  the  committee  meets  to  elect  a  chairperson.  By  charter,  the  review 
committee  chairperson  is  the  single  focal  point  between  the  review  committee  and  the  range 
under  review  for  the  duration  of  the  review  process. 
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Figure  2-2.  RCS  Certification  Process  and  Timeline. 


Once  the  review  committee  has  been  formed,  the  review  timeline  officially  starts,  and  the 
review  moves  into  the  Off- Site  Review  Phase.  The  range  sends  a  copy  of  the  range  book  to  each 
review  committee  member.  The  review  committee  has  three  months  to  review  the  range  book, 
and  grade  it  in  accordance  with  the  evaluation  criteria.  The  chairperson  gathers  the  individual 
grades  and  issues  “composite”  or  committee  grades  for  each  evaluation  criterion.  At  the  end  of 
the  three-month  off-site  review,  the  chairperson  transmits  the  committee’s  “mid-review” 
assessment  to  the  range.  The  mid-review  assessment  will  likely  reveal  areas  where  the  reviewers 
require  additional  documentation,  clarification,  or  improvement  in  some  or  all  range  book 
sections.  Also,  about  30  percent  of  the  evaluation  criteria  are  initially  ungraded  by  the  review 
committee  as  they  are  evaluated  later  during  the  on-site  staff  interviews.  If  the  review  committee 
identifies  any  marginal  or  unsatisfactory  range  book  sections,  the  review  committee  must  include 
positive  corrective  comments  showing  the  range  how  to  reach  satisfactory  compliance  in  each 
noncompliant  area. 

Next,  the  certification  review  process  moves  into  the  Feedback  and  Update  Phase.  Once 
the  range  receives  the  mid-review  report,  the  range  responds  to  the  feedback  within  14  days  and 
is  then  given  a  minimum  of  60  days  to  accomplish  range  book  revisions  and  prepare  for  the  on¬ 
site  review.  (Naturally,  more  time  may  be  requested  at  any  time  in  the  process  by  the  range 
under  review,  if  needed.)  The  range  makes  every  effort  to  correct  all  high  priority  deficiencies 
identified  by  the  review  committee.  Such  corrections  are  normally  accomplished  by  adding 
written  revisions  to  the  range  book  to  address  specific  deficiencies.  The  range  also  prepares  for 
the  upcoming  review  committee  “on-site”  visit.  This  on-site  visit  generally  addresses  three 
areas:  (1)  the  overall  range  organization  and  technical  operations;  (2)  deficient  areas  and  the 
steps  the  range  has  taken  in  the  last  60  days  to  correct  them;  (3)  staff  interviews  and  the  “on-site” 
audit  questions  from  the  evaluation  criteria. 
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All  on-site  questions  are  known  in  advanee.  The  on-site  interviewers  will  not 
deviate  or  add  additional  on-site  questions  from  those  present  in  the  evaluation  eriteria. 


The  final  phase  is  the  On-Site  Visit  and  Certifieation  Phase.  During  this  period,  the 
review  committee  typically  spends  2-4  days  at  the  range  for  an  on-site  visit  of  the  facility.  In 
addition  to  going  over  the  formal  changes  and  revisions  to  the  range  book  described  previously, 
it  provides  an  opportunity  for  the  reviewers  to  get  a  “real  world”  feel  for  the  day-to-day  range 
conditions  and  environment.  During  this  period,  the  remaining  “on-site”  staff  interview 
questions  are  asked,  and  any  final  questions  and  comments  from  the  committee  are  addressed. 

At  the  conclusion  of  the  on-site,  the  review  committee  is  required  to  give  the  range  an  “out- 
brief’  of  the  new  composite  evaluation  scores.  Under  most  circumstances,  if  the  range 
successfully  accomplishes  the  required  range  book  revisions  in  advance,  the  range  “passes”  and 
becomes  certified  at  this  point.  If,  however,  the  review  committee  requires  closure  on  a  few 
evaluation  criteria,  the  range  simply  resubmits  final  corrections  to  the  committee.  Once  the 
range  has  met  the  sufficient  number  of  evaluation  criteria  to  “pass,”  the  review  committee  issues 
a  final  report  to  the  RCC/SMSG  Chair  with  a  copy  to  the  range.  This  report  provides  the  final 
composite  grades  and  concludes  with  a  statement  that  the  range  has  passed  sufficient  criteria  to 
become  “Z-540  RCS  certified.”  The  final  range  book  evaluation  report  issued  by  the  review 
committee  also  becomes  a  permanent  part  of  the  range  book. 

2.4  Organizing  the  RCS  Range  Book  to  Comply  with  the  Z-540  Evaluation  Criteria 

The  following  information  provides  suggested  guidelines  for  the  construction  of  the  RCS 
Range  Book.  The  goal  of  this  guidance  is  to  maintain  consistency  in  the  layout  of  the 
documentation  from  range  to  range,  thus  reducing  the  time  needed  to  review  and  assess 
compliance  to  the  Z-540  standard.  Although  Volume  I  of  this  document  provides  a  detailed 
description  of  the  layout  of  a  typical  range  book,  there  are  several  lessons  learned  in  the 
construction  of  the  first  three  range  books  that  are  worth  sharing.  We  hope  range  managers  pay 
attention  to  these  lessons  so  that  ranges  desiring  certification  in  the  future  do  not  repeat  the  same 
mistakes  encountered  by  AFRL  and  NAWC-AD  during  the  DoD  RCS  Demonstration  Program. 

As  stated  many  times  earlier,  the  main  goal  of  the  range  book  is  to  create  and  organize 
the  RCS  system  documentation  into  a  format  independent  of  those  used  by  the  RCS  ranges 
performing  the  measurements.  While  there  is  a  great  deal  of  latitude  given  to  the  range 
organizing  the  information,  we  must  be  mindful  of  the  time  constraints  of  the  volunteer  range 
book  reviewers.  The  ACR  and  ATR  had  the  unfortunate  disadvantage  of  having  their  range 
books  written  before  the  evaluation  criteria  were  formally  released.  As  a  result,  spread 
throughout  their  range  books  were  many  pages  of  general  information  that  answered  multiple 
evaluation  criteria.  Because  they  wrote  the  books,  their  internal  reviewers  found  the  answers  to 
the  evaluation  criteria  in  minutes.  When  the  third  party  certification  review  committee  tried  to 
find  the  same  answers,  it  frequently  took  them  hours.  This  meant  that  the  review  committee  had 
to  read  and  reread  many  range  book  sections  to  gather  and  evaluate  all  the  information  present  in 


2-5 


the  range  books.  It  also  meant  that  AFRL  and  NAWC  had  many  mid-review  revisions  to 
accomplish,  since  their  range  books  were  not  organized  in  exact  line  with  the  evaluation  criteria. 

Learning  from  these  mistakes,  the  46  Test  Group  (RVUMS)  organized  their  range  book 
draft  much  more  efficiently.  The  RVUMS  range  book  sections  displayed  specific  chapters  of 
information  directly  in  parallel  with  the  evaluation  criteria  questions.  The  RVUMS  range  book 
listed  each  evaluation  criteria,  followed  immediately  by  the  information  needed  to  address  the 
criteria.  By  organizing  in  this  manner,  the  46TG  made  their  range  book  sections  considerably 
shorter  in  total  length,  and  much  more  focused  to  the  point  of  the  evaluation  criteria.  This  subtle 
change  greatly  reduced  the  time  needed  by  the  third  party  reviewers  to  evaluate  the  range  book. 
(The  overall  RVUMS  range  book  certification  review  was  ongoing  at  time  this  report  went  to 
press,  although  preliminary  feedback  from  the  RVUMS  reviewers  indicate  that  the  information  is 
much  better  organized  than  the  first  two  range  books  from  ACR  and  ATL.) 

2.5  Third  Party  Review  of  the  Range  Book  -  AFRL  Experience 

Since  AFRL  was  the  original  architect  of  the  RCS  certification  review  program,  they  felt 
extremely  confident  that  ACR’s  range  book  would  sail  through  the  evaluation  process.  They  had 
performed  an  internal  “self  evaluation”  to  become  Z-540  “registered”  and  were  confident  they 
could  quickly  pass  any  review.  Imagine  their  collective  surprise  when  their  mid-review 
assessment  showed  less  than  53  percent  compliance  on  the  140  evaluation  criteria.  After  the 
initial  shock  of  the  poor  assessment  wore  off,  they  carefully  examined  where  they  had  gone 
wrong  and  set  about  to  make  the  required  revisions.  Their  first  major  revelation  was  that  much 
of  their  range  information  was  disjointed  and  unfocussed.  They  immediately  set  about  revising 
the  high  priority  (must  pass)  sections  of  the  range  book  and  organizing  them  to  specifically 
address  the  evaluation  criteria.  Next,  they  drafted  a  response  to  the  review  committee  chair 
acknowledging  all  the  deficient  areas  and  their  plan  to  reach  closure.  Finally,  they  focused  their 
activities  to  succinctly  rewrite  the  deficient  sections  and  to  present  the  revisions  at  the  on-site 
review.  Since  the  review  committee  had  provided  excellent  and  thorough  comments,  AFRL 
knew  exactly  what  they  needed  to  do,  and  accomplished  all  the  revisions  within  the  allotted  60 
days.  The  on-site  review,  while  intense,  ended  on  a  totally  satisfactory  basis  for  both  the  range 
and  reviewers.  Their  final  compliance  grade  exceeded  93  percent,  with  no  unsatisfactory  criteria 
in  any  range  book  section. 

There  is  no  doubt  that  the  third  party  review  of  their  range  book  forced  them  to  concisely 
rewrite  many  of  their  core  processes.  While  difficult,  there  is  no  doubt  that  the  revised 
documentation  sections  were  more  readable  and  better  organized  for  both  the  committee  and  the 
AFRL  employees.  Once  AFRL  adopted  the  mantra  that  “the  reviewers  are  right,”  the  whole 
attitude  within  the  staff  shifted  dramatically  to  a  positive  outlook.  The  AFRL  staff  came  up  with 
succinct  and  innovative  methods  and  graphics  to  illustrate  points  they  spent  many  pages  of  text 
describing  in  their  original  range  book  draft,  making  critical  points  far  easier  for  their  technicians 
to  understand  and  comprehend.  In  addition,  since  their  primary  technicians  were  closely 
involved  in  the  range  book  revisions,  they  became  intimately  familiar  with  the  purpose  and 
content  of  the  range  book. 
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During  the  range  book  “on-site”  review  interviews,  the  review  committee  picked  up  on 
the  “whole  team”  cohesion.  The  review  committee  expressed  extreme  satisfaction  that  the 
quality  assurance  processes  and  purposes  of  the  range  book  were  understood  from  the  lowest 
technician  to  the  highest  range  technical  manager.  The  review  process  forced  AFRL  to  take  a 
much  harder  and  discerning  look  at  our  processes  and  documentation,  resulting  in  the  overall 
improvement  of  several  key  procedures.  Though  it  took  more  of  an  effort  than  they  originally 
anticipated,  there  is  no  doubt  that  the  ACR  is  a  better  organization  having  gone  through  the  Z- 
540  RCS  certification  process. 

2.6  The  Range  Book  Reviewers 

The  range  book  reviewers  are  volunteer  experts  from  government,  academia,  and 
industry  who  have  documented  experience  in  RCS  measurements.  The  entire  certification 
process  is  centered  on  the  concept  of  a  “third  party  peer  review.”  Consistent  with  ISO  practices, 
the  certification  reviewers  are  specialists  who  understand  the  technical  details  of  RCS  ranges.  In 
the  DoD  RCS  Demonstration  Program,  no  DoD  employees  were  used  as  reviewers  to  avoid  the 
appearance  of  conflict  of  interest,  though  many  such  RCS  experts  certainly  reside  in  the 
government.  Reviewers  for  the  three  DoD  demonstration  ranges  (Figure  2-1.)  came  from  Boeing 
Phantom  Works,  Northrop  Corporation,  EG&G  Corporation,  Syracuse  Research  Corporation, 
Mission  Research  Corporation  and  the  National  Institute  of  Standards  and  Technology  (NIST). 
The  reviewers  represented  themselves  as  RCS  technical  experts,  rather  than  representatives  of 
their  corporations  or  organizations. 

The  process  of  becoming  an  RCS  range  book  reviewer  is  straightforward.  Those 
interested  in  becoming  reviewers  must  prepare  two  documents:  (1)  a  one-page  resume 
describing  RCS  design  experience  and/or  a  minimum  of  five  years  of  relevant  RCS  measurement 
experience,  and  (2)  a  corporate  letter  of  commitment  from  the  applicant’s  employer  stating  that 
the  applicant  may  participate  in  two  range  reviews  in  three  years,  and  that  release  time  will  be 
granted  the  applicant  to  support  the  on-site  range  reviews.  This  resume  and  corporate 
commitment  letter  is  transmitted  to  the  RCC/SMSG  Radar  Committee  (as  of  the  date  of  this 
publication  the  Chair  is  Mr.  John  Lewis  of  NAWC-AD,  tel.#  301-342-1193,  e-mail 
lewisja@navair.navv.mil.)  The  Radar  Committee  Chair  then  reviews  the  resume  and  submits  it 
to  the  RCC/SMSG  Radar  Committee  for  evaluation.  Upon  acceptance  of  the  applicant 
reviewer’s  qualifications,  the  individual’s  name  is  added  to  the  pool  of  RCS  review  team 
candidates,  as  discussed  previously  in  this  report.  A  reviewer  is  selected  for  a  specific  range 
book  review  based  on  two  main  criteria:  (1)  the  availability  of  the  reviewer  for  a  specific  review 
period,  and  (2)  the  lack  of  any  perceived  or  financial  conflict  of  interest  between  the  reviewer 
and  the  range  under  review. 

Finally,  one  may  ask,  “Why  become  a  reviewer?”  For  ranges  anticipating  RCS 
certification  in  the  future,  it  is  strongly  recommended  that  at  least  one  person  from  that 
organization  volunteer  to  be  a  reviewer  for  another  range.  To  avoid  inefficient,  small  dollar 
money  transfers  between  companies  in  this  program,  we  have  operated  the  program  on  the  quid- 
pro-quo  foundation  that  if  a  range  wants  to  be  reviewed  (certified),  it  should  be  prepared  to 
review  another  range.  In  this  manner,  solid  RCS  experience  and  best  practices  will  migrate 
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throughout  the  RCS  measurement  industry,  raising  both  the  overall  quality  of  measurement 
practices  while  instilling  “generally  accepted  measurement  practices”  throughout  the  RCS 
measurement  industry. 
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CHAPTER  3 


THE  DOD  RCS  DEMONSTRATION  PROGRAM  -  NAWC-AD  PERSPECTIVE 


3.1  General 

This  chapter  describes  the  Naval  Air  Warfare  Center-Aircraft  Division  (NAWC-AD) 
portion  of  the  DoD  RCS  Demonstration  Program,  focusing  on  the  NAWC-AD  Atlantic  Test 
Range  (ATR)  range  book  review  processes,  outcomes,  and  lessons  learned. 

To  facilitate  evaluation  of  the  ATR  range  book,  a  set  of  evaluation  criteria  based  on  those 
listed  in  Z-540  (ISO  25)  were  developed.  Upon  receiving  word  that  the  respective  range  books 
were  ready  for  evaluation,  the  RCC/SMSG  chose  evaluation  teams  from  industry  and  the 
government  to  assess  conformance  to  Z-540  by  the  Air  Force  Research  Lab  (AFRL),  NAWC- 
AD,  and  NRTF.  Two  of  the  three  range  book  assessments  are  now  complete.  This  chapter 
discusses  the  assessment  of  the  ATR  an  outdoor  dynamic  range. 

3.2  Atlantic  Test  Range  (ATR)  Description 

The  Atlantic  Test  Range  (ATR)  is  a  dynamic  in-flight  test  facility  performing  RCS 
measurements  on  aircraft  for  U.S.  military,  foreign,  and  civilian  customers.  The  ATR  is  part  of 
the  Atlantic  Ranges  and  Facilities,  a  department  of  the  Naval  Air  Warfare  Center- Aircraft 
Division.  Located  at  the  mouth  of  the  Patuxent  River  overlooking  Chesapeake  Bay  and 
controlling  2400  square  miles  of  restricted  airspace,  the  facility  is  able  to  conduct  measurements 
in  a  low-clutter  environment. 

The  ATR  uses  a  coherent  RCS  measurement  system,  the  Advanced  Dynamic  Aircraft 
Measurement  System  (ADAMS),  to  obtain  real-time  data  for  Doppler  signatures,  down-range 
profiles,  inverse  synthetic  aperture  radar  (ISAR)  images,  as  well  as  whole-body  RCS  data. 
ADAMS  is  also  used  to  collect  chaff  RCS,  jammer-skin  ratio  (J/S),  decoy-skin  ratio  (D/S),  and 
antenna  patterns. 

The  ADAMS  collects  data  at  153.5  MHz,  230  MHz,  425  MHz  and  2-18  GHz,  and  at  35 
GHz  non-coherently.  Up  to  80  frequencies  can  be  simultaneously  transmitted  at  pulse  rates 
ranging  from  a  few  samples  per  second  to  a  maximum  of  200,000  samples  per  second.  These 
data  are  statistically  processed  off-line  to  produce  probability  distribution  functions;  minimum 
and  maximum  values,  and  mean,  variance,  standard  deviation  and  percentile  data. 

3.3  ATR  Initial  Range  Book  Evaluation 

At  the  1999  RCS  Range  Certification  Conference,  the  RCC/SMSG  Radar  Committee 
presented  the  draft  RCS  certification  process  to  the  RCS  measurement  community.  As  described 
in  Chapter  2  of  this  report,  this  process  was  meant  to  be  in  draft  form  until  its  evaluation  was 
completed  in  the  DoD  RCS  Demonstration  Program.  References  [8,  9,  10]  show  the  timeline 
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and  discuss  the  rationale  behind  the  DoD  demonstration  program.  At  the  conclusion  of  the 
demonstration  program,  the  actual  certification  teams  and  review  team  members  would 
recommend  minor  modifications  to  the  draft  certification  process,  so  the  RCC/SMSG  could 
make  final  changes  to  the  processes  before  industrial  certification  implementation  began  in  2001. 
As  will  be  seen  later  in  this  paper,  this  philosophy  worked  so  well  that  less  than  five  percent  of 
the  evaluation  criteria  or  processes  were  changed  at  the  conclusion  of  the  demonstration 
program. 

The  timeline  for  the  RCS  certification  review  process  called  for  an  initial  three-month 
off-site  evaluation  period,  during  which  time  the  evaluation  team  would  review  the  submitted 
range  books  and  evaluate  them  for  compliance  with  Z-540.  After  this  three-month  evaluation 
period,  the  range  would  normally  be  given  approximately  60  days  to  correct  any  deficiencies 
found  during  the  initial  review.  Next,  the  evaluation  team  would  conduct  an  on-site  review  of 
the  range  and  look  at  corrective  actions  from  the  initial  review,  interview  range  personnel, 
observe  calibration  activities,  and  the  like.  If  the  on-site  review  is  executed  properly,  the  review 
team  and  the  range  should  quickly  reach  closure  on  any  outstanding  evaluation  criteria. 

As  previously  discussed,  the  chair  of  the  RCC/SMSG  gathered  both  the  AFRL  (ACR) 
and  NAWC-AD  (ATR)  evaluation  teams  from  a  pool  of  qualified  volunteer  RCS  measurement 
experts.  Each  team  was  composed  of  two  industry  experts  plus  one  representative  from  the  U.S. 
Department  of  Commerce  (National  Institute  of  Standards  and  Technology  -NIST). 

The  evaluation  teams  made  use  of  a  set  of  over  140  evaluation  criteria  [8]  derived  from 
sample  evaluation  criteria  contained  in  the  Z-540  handbook  and  rewritten  to  specifically  address 
topics  relevant  to  RCS  measurement  ranges.  As  described  in  Chapter  2,  when  the  AFRL  and 
NAWC-AD  ranges  were  assembling  their  initial  range  book  submissions,  they  did  not  have  the 
initial  evaluation  criteria  as  a  reference.  Therefore,  their  initial  range  book  submissions  did  not 
directly  answer  the  criteria.  Instead,  they  submitted  their  entire  range  documentation  sets  for 
evaluation.  This  posed  a  major  logistical  problem  for  the  evaluators  as  they  were  performing  this 
task  on  a  part-time  basis. 

In  the  case  of  the  ATR  team,  this  meant  reviewing  over  560  pages  of  ATR  documentation 
spread  over  three  printed  volumes.  Since  the  ATR  and  ACR  range  books  were  originally 
designed  to  be  totally  web-based  electronic  books,  the  review  process  became  complicated 
because  the  range  books  were  not  initially  designed  to  be  printed.  As  a  result,  areas  that  were 
logically  linked  in  their  native  web  format  were  not  linked  on  paper  at  all.  For  the  review  team 
members,  this  meant  that  chasing  down  references  in  the  text  tended  to  be  very  laborious  and 
time  consuming. 

In  general,  the  ATR  range  book  review  team  found  the  initial  written  submissions  to  be 
poorly  put  together.  Since  ATR  did  not  use  the  evaluation  criteria  as  a  guide,  the  submission 
evolved  into  an  ad  hoc  collection  of  (sometimes  hastily  put  together)  discussions  on  calibration 
and  measurement  procedures,  along  with  checklists  taken  out  of  context.  Forcing  a  hard-copy 
printout  resulted  in  poor  technical  editing  (due  to  the  time  constraints),  and  irregular  hard-to-read 
figures  and  graphs.  Everyone  realized  after  the  books  were  delivered  that  it  truly  did  not 
represent  the  quality  the  team  members  were  accustomed  to  seeing  from  ATR. 
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Eventually,  the  initial  RCS  certification  review  team  mid-review  report  was  completed. 
The  result  was  a  21 -page  document  submitted  to  ATR  detailing  the  various  deficiencies  found  in 
the  documentation,  as  well  as  various  requests  for  clarification.  As  required,  after  each  listed 
deficiency  the  review  team  included  suggestions  on  how  the  problem  could  be  solved.  Thus,  the 
mid-review  report  was  used  as  an  improvement  tool,  not  a  hammer.  The  report  also  included  a 
mid-review  grade  which  reflected  the  percentage  of  evaluation  criteria  the  review  team  felt  ATR 
met.  It  must  be  noted  here  that  this  was  merely  an  initial  findings  report,  again  with  the  purpose 
of  allowing  ATR  the  opportunity  to  explain  and/or  rectify  any  problems  prior  to  the  on-site 
review. 


Put  bluntly,  the  report  was  eye  opening  to  ATR  personnel.  After  all,  ATR  had  put 
considerable  time  and  effort  into  their  range  book  and  felt  they  had  all  bases  covered.  They  were 
aware  that  some  issues  were  still  open;  however,  the  magnitude  of  the  problems  uncovered  by 
the  evaluation  team  was  unexpected.  Given  the  initial  set  of  grades  received,  which  were 
significantly  below  the  passing  threshold,  the  review  team  mid-review  report  could  easily  have 
been  interpreted  by  the  ATR  staff  as  a  slap  in  the  face.  To  their  great  credit,  ATR  viewed  the 
report  objectively  and  accepted  the  criticism  at  face  value,  resulting  in  a  sustained  and 
cooperative  recovery  effort  of  the  highest  quality. 

3.4  ATR  Response  to  the  Initial  Review  Committee  Evaluation 

Once  the  initial  shock  of  the  low  scores  was  overcome,  the  ATR  team  immediately 
responded  to  the  report.  The  response  consisted  of  two  items :  a  line-by-line  response  to  the 
initial  findings,  and  a  reformatted  submission  that  closely  followed  the  evaluation  criteria. 

The  line-by-line  responded  to  each  and  every  question  posed  in  the  initial  findings.  The 
individual  responses  in  many  cases  were  simple  acknowledgments  of  the  questions  and  the 
implementation  of  the  suggested  solutions.  A  large  number  of  criteria  required  significant 
explanation  of  what  the  original  submission  discussed,  why  and  how  the  new  submission 
differed,  and  the  evaluation  criteria. 

Many  questions  posed  in  the  initial  findings  caused  the  ATR  team  to  re-examine  its 
procedures  (which,  of  course,  was  the  point  of  the  exercise).  Sometimes  this  re-examination 
resulted  in  a  change  in  a  procedure,  at  other  times  the  result  was  a  task  for  future  study.  In  the 
first  instance,  for  example,  ATR  decided  to  use  the  average  sphere  diameter  from  their  sphere 
shipments  in  the  Mie  equation  calculations,  rather  than  a  nominal  value. 

The  reformatted  submission  took  the  list  of  evaluation  criteria  and  addressed  them 
sequentially.  It  was  a  much  more  succinct  and  direct  method  of  ensuring  compliance  with  Z- 
540,  and  is  being  recommended  as  a  template  for  future  ranges  to  use  in  submitting  their  range 
books  for  review.  As  mentioned  in  Chapter  2,  the  RVUMS  range  book  submission  has  already 
incorporated  these  lessons  learned. 


3-3 


3.5 


The  ATR  On-Site  Review 


The  next  critical  portion  of  the  RCS  certification  review  process  is  the  on-site  review 
phase.  This  gives  the  range  under  review  the  opportunity  to  showcase  their  range  and  procedures 
to  the  review  team.  It  also  offers  the  opportunity  to  discuss  the  required  improvements  and 
changes  to  the  ATR  range  book  brought  about  by  the  initial  RCS  mid-review  report.  The 
certification  review  team  also  has  the  opportunity  to  see  the  day-to-day  operations  at  the  range, 
to  interview  range  personnel  (some  30  percent  of  the  evaluation  criteria  are  “on-site”  questions), 
and  review  the  changes  made  to  the  range  book. 

The  review  team  made  a  visit  to  ATR  during  the  last  week  of  April  2000  (following  a 
similar  trip  to  AFRL).  For  consistency  in  the  demonstration  program,  it  was  decided  that  both 
the  AFRL  and  NAWC-AD  review  teams  would  attend  each  other’s  respective  on-site  reviews.  It 
was  felt  that  this  would  provide  a  level  of  grading  continuity,  as  well  as  an  extra  set  of 
viewpoints  for  those  reviewers  in  advisory  (non- voting)  roles. 

During  the  on-site  ATR  review,  range  personnel  conducted  a  comprehensive  tour  of  the 
range,  which  gave  the  review  team  a  feel  for  the  extreme  complexity  of  flight  test  operations. 
Range  personnel  also  conducted  a  calibration  sphere  launch  and  tracked  it  out  to  the  limits  of 
their  system.  The  review  team  had  the  opportunity  to  meet  with  ATR  management  and  gauge 
their  knowledge  of  and  support  for  the  range  book  process.  It  was  immediately  obvious  to  the 
team  that  ATR  management  was  fully  supportive  of  the  process  and  was  looking  at  ways  to 
apply  the  certification  and  quality  assurance  ideas  to  other  facilities  at  NAWC-AD. 

Finally,  the  ATR  measurement  team  simulated  a  measurement  by  replaying  an  actual 
flight  from  tape  and  discussing  the  procedures  as  the  flight  proceeded.  Flight  paths  were 
reproduced  and  discussed,  flight  controller  duties  were  outlined,  and  data  collection  tasks  and 
procedures  were  reviewed.  The  review  team  came  away  from  the  simulation  extremely 
impressed  with  the  way  complex  procedures  were  executed  and  problems  were  resolved,  and 
with  the  overall  professionalism  of  the  measurement  team. 

After  the  orientation  segments,  the  real  on-site  review  work  began.  The  review  team  sat 
down  with  several  ATR  range  representatives  and  discussed  the  revised  range  book  in  detail.  As 
mentioned,  ATR  had  extensively  re-worked  their  range  book  submission.  Even  though  the 
review  team  had  the  opportunity  to  review  the  revisions  prior  to  their  arrival  at  ATR,  the  changes 
were  substantial  enough  to  require  thorough  on-site  discussion  and  review.  The  revised  ATR 
range  book  review  was  much  easier  the  second  time  around  since  ATR  now  used  the  evaluation 
criteria  as  a  guide. 

As  with  any  committee  activity,  the  process  did  not  go  entirely  smoothly.  Even  with  the 
evaluation  criteria  in  hand,  there  were  differing  review  team  interpretations  of  the  meaning 
and/or  applicability  of  several  of  the  evaluation  criteria.  These  differences  had  to  be  discussed 
and  common  interpretations  hammered  out.  Eurthermore,  there  was  considerable  discussion 
(especially  in  the  closed-door  team-only  sessions)  as  to  what  was  required  to  satisfy  several 
critical  criteria.  The  diversity  of  the  review  team  meant  that  the  severity  with  which  the 
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evaluation  criteria  were  interpreted  varied  among  the  review  team  members.  The  composite 
grading  concept  used  in  the  review  process  resolved  some  disagreements  if  two  of  the  three 
review  team  members  felt  the  criteria  were  sufficiently  satisfied.  Despite  these  splits,  the  three 
review  team  members  were  in  complete  agreement  on  most  of  the  final  scores. 

Discussions  dealing  with  these  exit  criteria  were  probably  the  longest  and  most  difficult 
of  all,  though  in  the  end  satisfactory  solutions  were  adopted.  The  review  team  commitment  was 
substantial  during  the  on-site  review.  Team  members  often  arrived  with  the  earliest  arriving 
ATR  personnel  and  left  when  the  building  was  vacated  in  the  evening.  In  the  end,  ATR  was 
pleased  to  report  that  the  team  was  able  to  come  to  a  unanimous  consensus  on  the  evaluation.  As 
mentioned,  the  initial  evaluation  returned  a  grade  substantially  below  that  necessary  to  pass. 

After  the  on-site  review,  the  scores  in  all  categories  rose  precipitously.  Although  Priority  2  and  3 
sections  passed  with  large  margins,  ATR  initially  fell  just  short  of  passing  in  a  couple  of  critical 
Priority  1  sections. 

The  ATR  took  the  news  of  the  shortfalls  in  Priority  1  sections  in  stride.  Since  the  ATR 
range  manager  anticipated  the  shortfalls  identified  by  the  review  team,  ATR  was  already 
executing  a  solid  correction  strategy.  The  review  team  left  NAWC-AD  confident  ATR  would 
correct  enough  deficiencies  to  receive  a  passing  grade  by  the  4th  RCS  Conference,  held  20-22 
June  2000  at  Boulder,  CO). 

3.6  ATR  Wrap-Up  Actions 

During  the  ATR  on-site  review,  the  tight  interconnections  between  some  of  the  Z-540 
evaluation  criteria  became  obvious.  If  a  weakness  was  found  in  one  critical  process,  that 
weakness  tended  to  cascade  through  several  different  criteria.  On  the  positive  side,  that  also 
meant  a  single  fix  could  correct  several  criteria  needing  improvement. 

One  area  identified  during  both  the  AFRL  and  NAWC-AD  reviews  was  the  need  to  tie 
the  uncertainty  analysis  directly  to  calibration  measurements.  To  satisfy  this  requirement,  all 
parties  agreed  a  Report  of  Measurement  would  be  produced.  This  report  was  to  contain  a 
complete  measurement  of  two  objects  using  a  method  similar  to  that  in  [12],  wherein  a  known 
object  was  used  to  calibrate  the  measurement  system,  then  another  known  object  was  measured 
and  the  uncertainty  analysis  applied  [13].  Since  the  second  object  was  known,  the  resulting 
errors  should  fall  within  the  uncertainty  bounds  calculated  for  that  measurement.  The  report  had 
a  succinct  format,  allowing  a  good  deal  of  information  to  be  presented  in  a  small  number  of 
pages. 


The  ATR  and  the  ACR  both  completed  the  required  RCS  Report  of  Measurement  and 
submitted  the  results  to  the  review  team  for  consideration.  After  reviewing  the  ATR  report,  the 
RCS  certification  review  team  agreed  that  ATR  had  passed  sufficient  criteria  to  receive  a  passing 
certification  grade.  This  result  was  documented  in  the  final  report  and  awarded  during  the  June 
2000  RCS  Range  Certification  Conference  in  Boulder. 
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3.7  ATR  RCS  Certification  -  Lessons  Learned 


The  range  book  is  intended  to  help  each  range  organize  its  documentation  in  a  useful 
manner.  The  Z-540  provides  the  minimum  set  of  requirements  a  documentation  set  must  satisfy 
to  pass  the  RCS  certification  process.  The  review  committees  learned  that  a  solid  method  to 
organize  the  range  book  around  the  Z-540  evaluation  criteria  is  absolutely  essential.  The 
evaluation  criteria  in  Vol.  I  of  this  document  provides  the  necessary  template. 

As  mentioned  earlier,  the  most  important  lesson  learned  from  the  NAWC-AD 
demonstration  program  is  the  need  to  answer  the  evaluation  criteria  questions  directly  and 
concisely.  Although  the  initial  submissions  did  not  have  the  evaluation  criteria  from  which  to 
work;  future  ranges  will  have  them  available  as  an  open-book  test.  Since  the  evaluators  are  part- 
time  volunteers  and  have  their  own  jobs  to  perform,  direct,  concise  answers  to  the  evaluation 
criteria  are  not  only  helpful  but  critical  to  the  timely  completion  of  the  certification  review. 

Another  general  lesson  learned  is  the  need  for  a  printed  or  printable  copy  of  the 
submission  so  that  the  evaluator  can  physically  do  the  evaluation.  Hard  copies  also  eliminate 
problems  with  different  computer  platforms  and  make  note  taking  easier. 

The  Report  of  Measurement  [13]  is  also  a  critical  item  because  it  ties  together  the 
calibration  and  uncertainty  analysis  and  demonstrates  proficiency  in  both.  It  does  so  in  a  very 
compact  and  straightforward  format  which  is  available  from  the  AFRL  or  NAWC-AD 
demonstration  program  participants. 
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CHAPTER  4 


DOD  RCS  DEMONSTRATION  PROGRAM  SUMMARY 
AND  FUTURE  RCS  CERTIFICATION  REQUIREMENTS 


4.1  General 

The  primary  emphasis  of  this  program  is  to  improve  the  quality  and  repeatability  of  RCS 
data  acquired  at  DoD  RCS  ranges.  The  DoD  RCS  Demonstration  Program  is  designed  to  show 
both  industry  and  the  government  ranges  that  this  is  a  worthwhile,  value-added  quality  assurance 
initiative.  As  such,  many  industrial  RCS  measurement  facilities,  including  those  owned  by 
Boeing,  Northrop,  and  Lockheed,  are  now  actively  pursuing  RCS  range  certification.  Today, 
there  is  no  requirement  to  obtain  RCS  certification.  Beginning  on  1  January  2004,  however,  all 
industrial  RCS  measurement  facilities  supplying  deliverable  RCS  data  to  AFRL  will  be  required 
to  obtain  RCS  data  on  a  certified  RCS  measurement  range.  Thus,  the  voluntary  aspect  of  range 
certification  will  revert  to  a  requirement  within  three  years  for  much  of  the  RCS  measurement 
industry.  There  is  more  than  sufficient  time  for  most  ranges  to  create  a  range  book  in  accordance 
with  Z-540.  In  addition,  there  is  much  documentation  available  from  the  demonstration  program 
to  help  any  range  quickly  come  up  to  speed  on  the  certification  process.  Since  industry  is 
actively  working  to  comply,  it  only  makes  sense  for  the  remaining  U.S.  Army,  Navy,  and  Air 
Force  facilities  performing  RCS  measurements  to  comply  with  the  Z-540  certification 
requirements  as  well. 

4.2  Certifying  to  Z-540 

Certifying  an  RCS  range  to  the  Z-540  standard  is  an  excellent  approach  to  organizing, 
maintaining,  and  presenting  RCS  range  information.  Any  range  book  that  passes  the  third  party 
review  to  become  “certified”  will  produce  a  document  useful  to  the  range  and  range  customer 
alike.  In  addition,  it  will  permanently  capture  critical  process  information  that  is  retained  by  the 
range  even  as  personnel  turnover  or  are  assigned  to  other  duties. 

We  believe  that  with  the  overall  emphasis  on  quality  systems  and  ISO  9000  in  the 
commercial  sectors,  Z-540  (ISO  25)  is  an  appropriate  technical  standard  for  RCS  measurement 
facilities.  Certifying  RCS  ranges  to  this  standard  should  greatly  reduce  measurement  errors  and 
generally  improve  the  operational  efficiencies  of  all  RCS  ranges.  In  addition,  certification  will 
allow  a  range  to  assemble,  in  one  paper  document  or  on  an  internal  web-site,  all  the  relevant 
information  needed  to  operate  an  RCS  measurement  facility.  This  should  help  any  range  sustain 
its  measurement  quality  during  periods  of  personnel  turnover,  since  the  processes  and  procedures 
will  be  there  even  after  employees  depart.  In  light  of  career  mobility,  and  a  national  aging 
technical  workforce,  this  benefit  alone  should  be  a  powerful  incentive  for  any  organization  to  get 
and  stay  RCS  certified. 


4-1 


The  DoD  RCS  Demonstration  Program  has  already  shown  benefits  in  new  calibration 
techniques,  new  methods  of  uncertainty  analysis,  and  a  better  understanding  of  error  sources. 

The  project  has  also  resulted  in  a  workable  and  useful  standard  for  organizing  range 
documentation. 

4.3  RCS  Certification  Process 

Probably  the  most  surprising  aspect  of  the  DoD  RCS  Demonstration  Program  is  how 
robust  the  initial  draft  certification  process  really  was.  Despite  early  teething  problems  and  the 
need  to  train  the  RCS  review  teams  on  the  process,  in  the  end  the  reviewers  and  ranges  both 
agreed  the  process  was  fair,  impartial,  and  value-added.  In  addition,  a  post  examination  of  the 
review  process  (as  a  whole),  and  the  individual  evaluation  criteria  after  the  ACR  and  ATR 
reviews,  revealed  that  only  minor  changes  to  the  process  were  needed.  Therefore,  it  is  likely  that 
the  RCS  industrial  certification  program  will  be  nearly  identical  to  the  DoD  program  already 
completed. 

As  for  side  benefits,  both  ACR  and  ATR  personnel  remarked  how  much  easier  it  is  to 
find  often-used  information  within  the  structure  of  their  revised  range  books,  as  opposed  to  the 
prior  ad  hoc  random  access  methods.  With  the  measurement  community  becoming  older, 
smaller,  and  more  mobile,  the  resulting  consolidation  of  RCS  documentation  should  help 
alleviate  the  impact  of  training  and  personnel  retention  problems,  thereby  helping  each  range 
sustain  its  high-quality  measurement  efforts. 

As  DoD  is  now  paying  more  attention  to  ISO  9000-like  processes,  the  DoD  RCS 
Demonstration  Program  for  RCS  certification  is  a  relevant  way  for  each  range  to  preemptively 
hold  off  management  dictates  concerning  quality  processes.  The  RCS  review  process  works,  and 
the  results  make  creating  the  range  book  worth  the  effort.  It  is  our  collective  opinion  that  it  is 
time  for  this  RCS  certification  effort  to  become  the  norm.  Our  RCS  customers  will  only  expect 
quality  to  improve  in  the  future,  and  it  is  our  job  to  deliver  the  promise. 

4.4  Conclusion 

In  summary,  the  DoD  RCS  Demonstration  Program  participants  believe  that  the  program, 
as  executed,  was  entirely  successful  in  achieving  its  goals.  For  the  first  time  in  history,  an  RCS 
range  documentation  standard  exists,  and  it  is  flexible  enough  to  work  at  any  type  of  RCS  range. 
The  Z-540  standard  is  a  universally  understood  and  adopted  commercial  standard,  and  its 
applicability  to  RCS  has  been  conclusively  demonstrated.  The  three  RCS  ranges  participating  in 
this  demonstration  program  found  that  their  newly  documented  processes  and  procedures  were 
extremely  valuable  to  the  staff  and  range  managers  alike.  The  DoD  RCS  Demonstration 
Program  also  included  4  three-day  DoD  RCS  certification  conferences  hosted  by  the  National 
Institute  of  Standards  and  Technology,  Boulder,  CO  in  March  1997,  March  1998,  March  1999, 
and  June  2000.  The  proceedings  of  each  conference  have  been  widely  distributed,  resulting  in  an 
early  start  by  industry  in  the  RCS  certification  process.  Currently,  Boeing,  Lockheed,  Northrop, 
Ball  Aerospace,  and  General  Electric  are  following  the  AFRL  and  NAWC-AD  lead  in  creating 
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their  own  range  books.  This  industrial  buy-in  would  not  be  possible  in  this  budget-limited 
environment  if  industry  did  not  see  the  collective  wisdom  and  payoff  in  RCS  range  certification. 

There  is  nothing  magic  about  the  Z-540  standard  that  restricts  it  to  radio  frequency  RCS 
ranges.  Its  processes  and  procedures  are  general  enough  to  cover  all  types  of  scientific 
measurements,  including  laser  cross  section,  infrared  emissions  and  emissivity  testing,  radio 
frequency  property  of  materials  testing,  antenna  testing,  pressure  and  temperature  testing,  and  so 
forth.  If  government  laboratories  and  measurement  facilities  hope  to  continue  to  sustain  the 
finest  test  capabilities  possible  for  supporting  DoD’s  future  weapons  system  test  and  evaluation 
requirements,  the  Z-540  standard  should  be  looked  at  in  a  broader  context  for  application  to  these 
other  diverse  measurement  problems.  It  is  clear  to  the  RCC/SMSG  Radar  Committee  that  this 
standard  may  revolutionize  the  way  DoD  maintains  and  documents  its  in-house  test  capabilities. 
The  committee  strongly  recommends  that  the  RCC  look  for  other  opportunities  for  Z-540  to 
establish  and  document  quality  technical  measurement  systems  for  future  U.S.  military  hardware. 
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dedicated  group  are  to  be  highly  commended. 

A  portion  of  this  project  was  supported  by  the  Air  Porce  Information  Warfare  Center 
under  contract  P4I62I-97-D5200  and  by  several  additional  contracts  and  subcontracts  sponsored 
by  AERE  and  NAWC-AD.  Our  appreciation  to  these  organizations  for  this  invaluable 
assistance. 
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